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PATENT 

Attorney Docket No. 2307U-553 



BEDDED PROGRAM OBJECTS IN 
DISTRIBUTED HYPERMEDIA SYSTEMS 



Notice Regarding Copyrighted Material 
A portion of the disclosure of this patent document 
contains material which is subject to copyright protection. 
The copyright owner has no objection to the facsimile 
reproduction by anyone of the patent document or the patent 
disclosure as it appears in the Patent and Trademark Office 
file or records, but otherwise reserves all copyright rights 
whatsoever . 

BACKGROUND OF THE INVENTION 
This invention relates generally to manipulating 
data in a computer network, and specifically to retrieving, 
presenting and manipulating embedded program objects in 
distributed hypermedia systems. 

Computer networks are becoming increasingly popular 
as a medium for locating and accessing a wide range of data 
from locations all over the world. The most popular global 
network is the Internet with millions of computer systems 
connected to it. The Internet has become popular due to 
widely adopted standard protocols that allow a vast 
interconnection of computers and localized computer networks 
to communicate with each other. Computer systems connected to 
a network such as the Internet may be of varying types, e.g., 
mainframes, workstations, personal computers, etc. The 
computers are manufactured by different companies using 
proprietary hardware and operating systems and thus have 
incompatibilities in their instruction sets, busses, software, 
file formats and other aspects of their architecture and 
operating systems. Localized computer networks connected to 
the Internet may be incompatible with other computer systems 
and localized networks in terms of the physical layer of 
communication including the specific hardware used to 
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implement the network. Also, different networks use 
differing, incompatible protocols for transferring information 
and are not able to communicate with each other without a 
translation mechanism such as a "gateway" . 
5 The Internet provides a uniform and open standard 

for allowing various computers and networks to communicate 
with each other. For example, the Internet uses Transfer 
Control Protocol/ Internet Protocol ( M TCP/IP M ) that defines a 
uniform packet-switched communication standard which is 
10 ultimately used in every transfer of information that takes 
place over the Internet. 

Other Internet standards are the HyperText 
Transmission Protocol ("HTTP") that allows hypertext documents 
to be exchanged freely among any computers connected to the 
15 i: Internet and HyperText Markup Language ("HTML") that defines 
yj the way in which hypertext documents designate links to 
! ^ information. See, e.g., Berners-Lee, T. J., "The world-wide 
J* web," Computer Networks and ISDN Systems 25 (1992). 
-P A hypertext document is a document that allows a 

20 : : user to view a text document displayed on a display device 

connected to the user's computer and to access, retrieve and 
:~J view other data objects that are linked to hypertext words or 
\$ phrases in the hypertext document. In a hypertext document, 
^3 the user may "click on," or select, certain words or phrases 
25 in the text that specify a link to other documents, or data 
objects. In this way, the user is able to navigate easily 
among data objects. The data objects may be local to the 
user's computer system or remotely located over a network. An 
early hypertext system is HyperCard, by Apple Computer, Inc. 
30 HyperCard is a standalone system where the data objects are 
local to the user's system. 

When a user selects a phrase in a hypertext document 
that has an associated link to another document, the linked 
document is retrieved and displayed on the user's display 
35 screen. This allows the user to obtain more information in an 
efficient and easy manner. This provides the user with a 
simple, intuitive and powerful way to "branch off" from a main 
document to learn more about topics of interest. 
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Objects may be text, images, sound files, video 
data, documents or other types of information that is 
presentable to a user of a computer system. When a document 
is primarily text and includes links to other data objects 
according to the hypertext format, the document is said to be 
a hypertext document. When graphics, sound, video or other 
media capable of being manipulated and presented in a computer 
system is used as the object linked to, the document is said 
to be a hypermedia document. A hypermedia document is similar 
to a hypertext document, except that the user is able to click 
on images, sound icons, video icons, etc., that link to other 
objects of various media types, such as additional graphics, 
sound, video, text, or hypermedia or hypertext documents. 

Fig. 1 shows examples of hypertext and hypermedia 
documents and links associating data objects in the documents 
to other data objects. Hypermedia document 10 includes 
hypertext 20, an image icon at 22, a sound icon at 24 and more 
hypertext 26. Fig. 1 shows hypermedia document 10 
substantially as it would appear on a user's display screen. 
The user is able to select, or "click" on icons and text on a 
display screen by using an input device, such as a mouse, in a 
manner well-known in the art. 

When the user clicks on the phrase "hypermedia," 
software running on the user's computer obtains the link 
associated with the phrase, symbolically shown by arrow 30, to 
access hypermedia document 14. Hypermedia document 14 is 
retrieved and displayed on the user's display screen. Thus, 
the user is presented with more information on the phrase 
"hypermedia." The mechanism for specifying and locating a 
linked object such as hypermedia document 14 is an HTML 
"element" that includes an object address in the format of a 
Uniform Resource Locator (URL) . 

Similarly, additional hypertext 26 can be selected 
by the user to access hypertext document 12 via link 32 as 
shown in Fig. 1. If the user selects additional hypertext 26, 
then the text for hypertext document 12 is displayed on the 
user screen. Note that hypertext document 12, itself, has 
hypertext at 28. Thus, the user can click on the phrase 



"hypermedia" while viewing document: 12 to access hypermedia 
document 14 in a manner similar to that discussed above. 

Documents, and other data objects, can be referenced 
by many links from many different source documents. Fig. 1 
shows document 14 serving as a target link for both documents 
10 and 12. A distributed hypertext or hypermedia document 
typically has many links within it that specify many different 
data objects located in computers at different geographical 
locations connected by a network. Hypermedia document 10 
includes image icon 22 with a link to image 16. One method of 
viewing images is to include an icon, or other indicator, 
within the text. 

Typically, the indicator is a very small image and 
may be a scaled down version of the full image. The indicator 
may be shown embedded within the text when the text is 
displayed on the display screen. The user may select the 
indicator to obtain the full image. When the user clicks on 
image icon 22 browser software executing on the user's 
computer system retrieves the corresponding full image, e.g., 
a bit map, and displays it by using external software called a 
"viewer." This results in the full image, represented by 
image 16, being displayed on the screen. 

An example of a browser program is the National 
Center for Super computing Applications (NCSA) Mosaic software 
developed by the University of Illinois at Urbana/ Champaign, 
Illinois. Another example is "Cello" available on the 
Internet at http://www.law.cornell.edu/. Many viewers exist 
that handle various file formats such as ".TIF," ".GIF," 
formats. When a browser program invokes a viewer program, the 
viewer is launched as a separate process. The view displays 
the full image in a separate "window" (in a windowing 
environment) or on a separate screen. This means that the 
browser program is no longer active while the viewer is 
active. By using indicators to act as place holders for full 
images that are retrieved and displayed only when a user 
selects the indicator, data traffic over the network is 
reduced. Also, since the retrieval and display of large 
images may require several seconds or more of transfer time 
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the user does not have to wait to have images transferred that 
are of no interest to the user. 

Returning | to Fig. 1, another type of data object is 
a sound object shown as sound icon 24 within the hypermedia 
5 document. When the user selects sound icon 24, the user's 

computer accesses sound data shown symbolically by data file 
40. The accessed sound data plays through a speaker or other 
audio device. 

As discussed above, hypermedia documents allow a 
10 user to access different data objects. The objects may be 

text, images, sound files, video, additional documents, etc. 
As used in this specification, a data object is information 
capable of being retrieved and presented to a user of a 
,^ computer system. Some data objects include executable code 
15 5 combined with data. An example of such a combination is a 
;*! "self -extracting" data object that includes code to "unpack" 
j: or decompress data that has been compressed to make it smaller 
before transferring. When a browser retrieves an object such 
as a self -extracting data object the browser may allow the 
20 « user to "launch" the self -extracting data object to 

™| automatically execute the unpacking instructions to expand the 
M. data object to its original size. Such a combination of 
^ executable code and data is limited in that the user can do no 
J more than invoke the code to perform a singular function such 
25 as performing the self -extraction after which time the object 
is a standard data object. 

Other existing approaches to embedding interactive 
program objects in documents include the Object Linking and 
Embedding (OLE) facility in Microsoft Windows, by Microsoft 
30 Corp., and OpenDoc, by Apple Computer, Inc. At least one 

shortcoming of these approaches is that neither is capable of 
allowing a user to access embedded interactive program objects 
in distributed hypermedia documents over networks. 

Fig. 2 is an example of a computer network. In Fig. 
35 2, computer systems are connected to Internet 100, although in 
practice Internet 100 may be replaced by any suitable computer 
network. In Fig. 2, a user 102 operates a small computer 104, 
such as a personal computer or a work station. The user's 
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computer is equipped with various components, such as user 
input devices (mouse, trackball, keyboard, etc.), a display 
device (monitor, liquid crystal display (LCD), etc.), local 
storage (hard disk drive, etc.), and other components. 
5 Typically, small computer 104 is connected to a larger 

computer, such as server A at 106. The larger computer may 
have additional users and computer, systems connected to it, 
such as computer 108 operated by user 110. Any group of 
computers may form a localized network. A localized network 
10 does not necessarily adopt the uniform protocols of the larger 
interconnecting network (i.e., Internet 100) and is more 
geographically constrained than the larger network. The 
localized network may connect to the larger network through a 
"gateway" or "node" implemented on, for example, a server. 
15 Internet 100 connects other localized networks, such 

jVj as server B at 120, which interconnects users 122, 124 and 126 
iy and their respective computer systems to Internet 100. 
J Internet 100 is made up of many interconnected computer 
p systems and communication links. Communication links may be 
20 — by hardwire, fiber optic cable, satellite or other radio wave 

si 

^ propagation , etc. Data may move from server A to server B 
u through any number of intermediate servers and communication 
%| links or other computers and data processing equipment not 
43 shown in Fig. 2 but symbolically represented by Internet 100. 
25 A user at a workstation or personal computer need 

not connect to the Internet via a larger computer, such as 
server A or server B. This is shown, for example/ by small 
computer 130 connected directly to Internet 100 as by a 
telephone modem or other link. Also, a server need not have 
30 users connected to it locally, as is shown by server C at 132 
of Fig. 2. Many configurations of large and small computers 
are possible. 

Typically, a computer on the Internet is 
characterized as either a "client" or "server" depending on 
35 the role that the computer is playing with respect to 

requesting information or providing information. Client 
computers are computers that typically request information 
from a server computer which provides the information. For 
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this reason, servers are usually larger and faster machines 
that have access to many data files, programs, etc., in a 
large storage associated with the server. However, the role 
of a server may also be adopted by a smaller machine depending 
5 on the transaction. That is, user 110 may request information 
via their computer 108 from server A. At a later time, server 
A may make a request for information from computer 108. In 
the first case, where computer 108 issues a request for 
information from server A, computer 108 is a "client" making a 

10 request of information from server A. Server A may have the 
information in a storage device that is local to Server A or 
server A may have to make requests of other computer systems 
to obtain the information. User 110 may also request 
information via their computer 108 from a server, such as 

15 ^ server B located at a remote geographical location on the 

UJ Internet. However, user 110 may also request information from 

ns 

'Z a computer, such as small computer 124, thus placing small 

■was 

4S computer 124 in the role of a "server." For purposes of this 
specification, client and server computers are categorized in 

20 terms of their predominant role as either an information 
requestor or provider. Clients are generally information 
"1 requestors, while servers are generally information providers, 
xj Referring again to Fig. 1, data objects such as 

distributed hypermedia documents 10, 12 and 14, image 16 and 

25 sound data file 40, may be located at any of the computers 

shown in Fig. 2. Since these data objects may be linked to a 
document located on another computer the Internet allows for 
remote object linking. 

For example, hypertext document 10 of Fig. 1 may be 

30 located at user 110 's client computer 108. When user 110 
makes a request by, for example, clicking on hypertext 20 
(i.e., the phrase "hypermedia"), user llO's small client 
computer 108 processes links within hypertext document 10 to 
retrieve document 14. In this example, we assume that 

35 document 14 is stored at a remote location on server B. Thus, 
in this example, computer 108 issues a command that includes 
the address of document 14 • This command is routed through 
server A and Internet 100 and eventually is received by server 
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B. Server B processes the command and locates document 14 on 
its local storage. Server 14 then transfers a copy of the 
document back co client 108 via Internet 100 and server A. 
After client computer 108 receives document 14, it is 
displayed so that user 110 may view it. 

Similarly; image object 16 and sound data file 40 
may reside at any of the computers shown in Fig. 2. Assuming 
image object 16 resides on server C when user 110 clicks on 
image icon 22, client computer 108 generates a command to 
retrieve image object 16 to server C. Server C receives the 
command and transfers a copy of image object 16 to client 
computer 108. Alternatively, an object, such as sound data 
file 40, may reside on server A so that it is not necessary to 
traverse long distances via the Internet in order to retrieve 
the data object. 

The Internet is said to provide an "open distributed 
hypermedia system." It is an "open" system since Internet 100 
implements a standard protocol that each of the connecting 
computer systems, 106, 130, 120, 132 and 134 must implement 
(TCP/IP) . It is a "hypermedia" system because it is able to 
handle hypermedia documents as described above via standards 
such as the HTTP and HTML hypertext transmission and mark up 
standards, respectively. Further, it is a "distributed" 
system because data objects that are imbedded within a 
document may be located on many of the computer systems 
connected to the Internet. An example of an open distributed 
hypermedia system is the so-called "world-wide web" 
implemented on the Internet and discussed in papers such as 
the Berners-Lee reference given above. 

The open distributed hypermedia system provided by 
the Internet allows users to easily access and retrieve 
different data objects located in remote geographic locations 
on the Internet. However, this open distributed hypermedia 
system as it currently exists has shortcomings in that todays 
large data objects are limited largely by bandwidth 
constraints in the various communication links in the Internet 
and localized networks, and by the limited processing power, 
or computing constraints, of small computer systems normally 



provided to most users. Large data objects are difficult to 
update at frame rates fast enough (e.g., 30 frames per second) 
to achieve smooth animation. Moreover, the processing power 
needed to perform the calculations to animate such images in 
real time does not exist on most workstations, not to mention 
personal computers. Today's browsers and viewers are not 
capable of performing the computation necessary to generate 
and render new views of these large data objects in real time. 

For example, the Internets open distributed 
hypermedia system allows users to view still images. These 
images are simple non-interactive two-dimensional images, 
similar to photographs. Much digital data available today 
exists in the form of high-resolution multi-dimensional image 
data (e.g., three dimensional images) which is viewed on a 
computer while allowing the user to perform real time viewing 
transformations on the data in order for the user to better 
understand the data. 

An example of such type of data is in medical 
imaging where advanced scanning devices, such as Magnetic 
Resonance Imaging (MRI) and Computed Tomography (CT) , are 
widely used in the fields of medicine, quality assurance and 
meteorology to present physicians, technicians and 
meteorologists with large amounts of data in an efficient way. 
Because visualization of the data is the best way for a user 
to grasp the data's meaning, a variety of visualization 
techniques and real time computer graphics methods have been 
developed. However, these systems are bandwidth- intensive and 
compute- intensive and often require multiprocessor arrays and 
other specialized graphics hardware to carry them out in real 
time. Also, large amounts of secondary storage for data are 
required. The expense of these requirements has limited the 
ability of researchers to readily exchange findings since 
these larger computers required to store, present and 
manipulate images are not available to many of the researchers 
that need to have access to the data. 

On the other hand, small client computers in the 
form of personal computers or workstations such as client 
computer 108 of Fig. 2 are generally available to a much 
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larger number of researchers. Further , it is common for these 
smaller computers to be connected to the Internet. Thus, it 
is desirable to have a system that allows the accessing , 
display and manipulation of large amounts of data, especially 
image data, over the Internet to a small, and relatively 
cheap, client computer. 

Due to the relatively low bandwidth of the Internet 
(as compared to today 1 s large data objects) and the relatively 
small amount of processing power available at client 
computers, many valuable tasks performed by computers cannot 
be performed by users at client computers on the Internet. 
Also, while the present open distributed hypermedia system on 
the Internet allows users to locate and retrieve data objects 
it allows users very little, if any, interaction with these 
data objects. Users are limited to traditional hypertext and 
hypermedia forms of selecting linked data objects for 
retrieval and launching viewers or other forms of external 
software to have the data objects presented in a 
comprehensible way. 

Thus, it is desirable to have a system that allows a 
user at a small client computer connected to the Internet to 
locate, retrieve and manipulate data objects when the data 
objects are bandwidth- intensive and compute- intensive. 
Further, it is desirable to allow a user to manipulate data 
objects in an interactive way to provide the user with a 
better understanding of information presented and to allow the 
user to accomplish a wider variety of tasks. 

SUMMARY OF THE INVENTION 
The present invention provides a method for running 
embedded program objects in a computer network environment. 
The method includes the steps of providing at least one client 
workstation and one network server coupled to the network 
environment where the network environment is a distributed 
hypermedia environment; displaying, on the client workstation, 
a portion of a hypermedia document received over the network 
from the server, where the hypermedia document includes an 
embedded controllable application; and interactively 
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controlling the embedded controllable application from the 
client workstation via communication sent over the distributed 
hypermedia environment. 

The present invention allows a user at a client 
computer connected to a network to locate, retrieve and 
manipulate objects in an interactive way. The invention not 
only allows the user to use a hypermedia format to locate and 
retrieve program objects, but also allows the user to interact 
with an application program located at a remote computer. 
Interprocess communication between the hypermedia browser and 
the embedded application program is ongoing after the program 
object has been launched. The user is able to use a vast 
amount of computing power beyond that which is contained in 
the user's client computer. 

In one application, high resolution three 
dimensional images are processed in a distributed manner by 
several computers located remotely from the user's client 
computer. This amounts to providing parallel distributed 
processing for tasks such as volume rendering or three 
dimensional image transformation and display. Also, the user 
is able to rotate, scale and otherwise reposition the 
viewpoint with respect to these images without exiting the 
hypermedia browser software. The control and interaction of 
viewing the image may be provided within the same window that 
the browser is using assuming the environment is a "windowing" 
environment. The viewing transformation and volume rendering 
calculations may be performed by remote distributed computer 
systems . 

Once an image representing a new viewpoint is 
computed the frame image is transmitted over the network to 
the user's client computer where it is displayed at a 
designated position within a hypermedia document. By 
transmitting only enough information to update the image, the 
need for a high bandwidth data connection is reduced. 
Compression can be used to further reduce the bandwidth 
requirements for data transmission. 

Other applications of the invention are possible. 
For example, the user can operate a spreadsheet program that 
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is being executed by one or more other computer systems 
connected via the network to the user's client computer. Once 
the spreadsheet program has calculated results, the results 
may be sent over the network to the user's client computer for 
5 display to the user. In this way, computer systems located 
remotely on the network can be used to provide the computing 
power that may be required for certain tasks and to reduce the 
data bandwidth by only transmitting results of the 
computations . 

10 Still other applications of the present invention 

are possible, as disclosed in the specification, below. 

\^ 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 illustrates examples of hypertext and 
15 3 hypermedia documents and links; 

Fig. 2 is an example of a computer network; 
j-; Fig. 3 r is an illustration of a computer system 

suitable for use with the present invention; 
H Fig. 4 is an illustration of basic subsystems in the 

20 computer system of Fig. 3; 

Fig. 5 is an illustration of an embodiment of the 
invention using a client computer,, server computer and a 
^ network; 

IS Fig. 6 shows another embodiment of the present 

25 invention using additional computers on the network; 

Fig. 7A is a flowchart of some of the functionality 
within the HTMLparse.c file; 

Fig. 7B is a flowchart of some of the functionality 
within the HTMLf ormat.c file; 
30 Fig. 8A is a flowchart of some of the functionality 

within the HTMLwidget.c file; 

Fig. 8B is a flowchart of some of the functionality 
within the HTML. c file; 

Fig. 9 is a screen display generated in accordance 
35 with the present invention; and 

Fig. 10 is a diagram of the various processes and 
data paths in the present invention. 
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DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT 
^iS^&^&fc Source code^microf iche Appendices A and B are 
provided to this specification* The source code should be 
consulted to provide details of a specific embodiment of the 
invention in conjunction with the discussion of the routines 
in this specification. The source code in Appendix A includes 
NCSA Mosaic version 2.4 source code along with modifications 
to the source code to implement the present invention. 
Appendix B includes source code implementing an application 
program interface. The source code is written in the "C" 
computer language to run on an X-Window platform. 

Fig. 3 is an illustration of a computer system 
suitable for use with the present invention. Fig. 3 depicts 
but one example of many possible computer types or 
configurations capable of being used with the present 
invention. Fig. 3 shows computer system 150 including display 
device 153 , display screen 155 , cabinet 157, keyboard 159 and 
mouse 161. Mouse 161 and keyboard 159 are "user input 
devices." Other examples of user input devices are a touch 
screen, light pen, track ball, data glove, etc. 

Mouse 161 may have one or more buttons such as 
buttons 163 shown in Fig. 3. Cabinet 157 houses familiar 
computer components such as disk drives, a processor, storage 
means, etc. As used in this specification "storage means" 
includes any storage device used in connection with a computer 
system such as disk drives, magnetic tape, solid state memory, 
bubble memory, etc; Cabinet 157 may include additional 
hardware such as input/output (I/O) interface cards for 
connecting computer system 150 to external devices such as an 
optical character reader, external storage devices, other 
computers or additional devices. 

Fig. 4 is an illustration of basic subsystems in 
computer system 150 of Fig. 3. In Fig. 4, subsystems are 
represented by blocks such as central processor 180, system 
memory 181 consisting of random access memory (RAM) and/or 
read-only memory (ROM), display adapter 182, monitor 183 
(equivalent to display device 153 of Fig. 3) , etc. The 
subsystems are interconnected via a system bus 184. 
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Additional subsystems such as a printer , keyboard, fixed disk 
and others are shown. Peripherals and input/output (I/O) 
devices can be connected to the computer system by, for 
example serial port 185. For example, serial port 185 can be 
used to connect the computer system to a modem for connection 
to a network or serial port 185 can be used to interface with 
a mouse input device. The interconnection via system bus 184 
allows central processor 180 to communicate with each 
subsystem and to control the execution of instructions from 
system memory 181 or fixed disk 186, and the exchange of 
information between subsystems. Other arrangements of 
subsystems and interconnections are possible. 

Fig* 5 is an illustration of an embodiment of the 
invention using a client computer, server computer and a 
network . 

In Fig. 5, client computer 200 communicates with 
server computer 204 via network 206. Both client computer 200 
and server computer 204 use a network protocol layer to 
communicate with network 206. In a preferred embodiment, 
network 206 is the Internet' and the network protocol layers 
are TCP/IP. Other networks and network protocols may be used. 
For ease of illustration, additional hardware and software 
layers are not shown in Fig. 5. 

Client computer 200 includes processes, such as 
browser client 208 and application client 210. In a preferred 
embodiment, application client 210 is resident within client 
computer 200 prior to browser client 208 's parsing of a 
hypermedia document as discussed below. In a preferred 
embodiment application client 210 resides on the hard disk or 
RAM of client computer 200 and is loaded (if necessary) and 
executed when browser client 208 detects a link to application 
client 210. The preferred embodiment uses the XEvent 
interprocess communication protocol to exchange information 
between browser client 208 and application client 210 as 
described in more detail, below. Another possibility is to 
install application client 210 as a "terminate and stay 
resident" (TSR) program in an operating system environment, 



such as X-Window. Thereby making access to application client 
210 much faster. 

Browser client 208 is a process that a user of 
client computer 200 invokes in order to access various data 
objects, such as hypermedia documents, on network 206. 
Hypermedia document 212 shown within client computer 200 is an 
example of a hypermedia document, or object, that a user has 
requested access to. In this example, hypermedia document 212 
has been retrieved from a server connected to network 206 and 
has been loaded into, e.g., client computer 200 ' s RAM or other 
storage device. 

Once hypermedia document 212 has been loaded into 
clxent computer 200, browser client 208 parses hypermedia 
document 212. In parsing hypermedia document 212, browser 
client 208 detects links to data objects as discussed above in 
the Background of the Invention section. In Fig. 5, 
hypermedia document 212 includes an embedded program link at 
214. Embedded program link 214 identifies application client 
212 as an application to invoke. In this present example, the 
application, namely, application client 210, resides on the 
same computer as the browser client 208 that the user is 
executing to view the hypermedia document. Embedded program 
link 214 may include additional information, such as 
parameters, that tell application client 210 how to proceed. 
For example, embedded program link 214 may include a 
specification as to a data object that application client 210 
is to retrieve and process. 

When browser client 208 encounters embedded program 
link 214, it invokes application client 210 (optionally, with 
parameters or other information) and application client 210 
executes instructions to perform processing in accordance with 
the present invention. 

An example of the type of processing that 
application client 210 may perform is multidimensional image 
visualization. Note that application client 210 is in 
communication with network 206 via the network protocol layer 
of client computer 200. This means that application client 
210 can make requests over network 206 for data objects, such 
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as multidimensional image objects. For example, application 
client 210 may request an object, such as object 1 at 216, 
located in server computer 204. Application client 210 may 
make the request by any suitable means. Assuming network 206 
is the Internet, such a request would typically be made by 
using HTTP in response to a HTML-style link definition for 
embedded program link 214. 

Assuming application client 210 has made a request 
for the data object at 216, server process 218 ultimately 
receives the request. Server process 218 then retrieves data 
object 216 and transfers it over network 206 back to 
application client 210. To continue with the example of a 
multidimensional visualization application, data object 216 
may be a three dimensional view of medical data for, e.g., an 
embryo • 

After application client 210 receives the 
multidimensional data object 216, application client 210 
executes instructions to display the multidimensional embryo 
data on the display screen to a user of the client computer 
200. The user is then able to interactively operate controls 
to recompute different views for the image data. In a 
preferred embodiment, a control window is displayed within, or 
adjacent to, a window generated by browser client 208 that 
contains a display of hypermedia document 212. An example of 
such display is discussed below in connection with Fig. 9. 
Thus, the user is able to interactively manipulate a 
multidimensional image object by. means of the present 
invention. In order to make application client 210 integral 
with displays created by browser client 208, both the browser 
client and the application client must be in communication 
with each other, as shown by the arrow connecting the two 
within client computer 200. The manner of communication is 
through an application program interface (API) , discussed 
below. 

Browser client 2 08 is a process, such as NCSA 
Mosaic, Cello, etc. Application client 210 is embodied in 
software presently under development called "VIS" and "Panel" 
created by the Center for Knowledge Management at the 



University of California , San Francisco, as part of the Doyle 
Group's distributed hypermedia object embedding approach 
described in "Integrated Control of Distributed Volume 
Visualization Through the World-Wide-Web, w by C. Ang, D. 
5 Martin, M. Doyle; to be published in the Proceedings of 

Visualization 1994, IEEE Press, Washington, D.C., October 
1994. 

Versions and descriptions of software embodying the 
present invention are generally available as hyperlinked data 
10 objects from the Visible Embryo Project's World Wide Web 
document at the URL address "HTTP: //visembryo. ucsf.edu/" . 

Another embodiment of the present invention uses an 
application server process executing on server computer 204 to 
^ assist in processing that may need to be performed by an 
15g external program. For example, in Fig. 5, application server 
jUJ 220 resides on server computer 204. Application server 220 
J* works in communication with application client 210 residing on 
*S client computer 200. In a preferred embodiment, application 
2 server 220 is called VRServer, also a part of Doyle Group's 
20 ^ approach. Since server computer 204 is typically a larger 
J computer having more data processing capabilities and larger 
u. storage capacity, application server 220 can operate more 
j efficiently, and much faster, than application client 210 in 
S executing complicated and numerous instructions. 
25 In the present example where a multidimensional 

image object representing medical data for an embryo is being 
viewed, application server 220 could perform much of the 
viewing transformation and volume rendering calculations to 
allow a user to interactively view the embryo data at their 
30 client computer display screen. In a preferred embodiment, 
application client 210 receives signals from a user input 
device at the user's client computer 200. An example of such 
input would be to rotate the embryo image from a current 
position to a new position from the user's point of view. 
35 This information is received by application client 210 and 
processed to generate a command sent over network 206 to 
application server 220. Once application server 220 receives 
the information in the form of, e.g., a coordinate 
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transformation for a new viewing position, application server 
220 performs the mathematical calculations to compute a new 
view for the embryo image. Once the new view has been 
computed, the image data for the new view is sent over network 
5 206 to application client 210 so that application client 210 
can update the viewing window currently displaying the embryo 
image. In a preferred embodiment, application server 220 
computes a frame buffer of raster display data, e.g., pixel 
values, and transfers this frame buffer to application client 
10 210. Techniques, such as data compression and delta encoding, 
can be used to compress the data before transmitting over 
network 206 to reduce the bandwidth requirement. 

It will be readily seen that application server 220 
can advantageously use server computer 204 's computing 

Q 

15 fQ resources to perform the viewing transformation much more 
W quickly than could application client 210 executing on client 
J computer 200. Further, by only transmitting the updated frame 
*p buffer containing a new view for the embryo image, the amount 
7\ of data sent over network 206 is reduced. By using 
20, appropriate compression techniques, such as, e.g., MPEG 
^ (Motion Picture Experts Group) or JPEG (Joint Photographic 
l! Experts Group) , efficient use of network 206 is preserved. 
"M Fig. 6 shows yet another embodiment of the present 

invention. Fig. 6 is similar to Fig. 5, except that 
25 additional computers 222 and 224 are illustrated. Each 

additional computer includes a process labeled "Application 
(Distributed)." The distributed application performs a 
portion of the task that an application, such as application 
server 220 or application client 210, perform. In the present 
30 example, tasks such as volume rendering may be broken up and 

easily performed among two or more computers. These computers 
can be remote from each other on network 206. Thus, several 
computers, such as server computer 204 and additional 
computers 222 and 224 can all work together to perform the 
35 task of computing a new viewpoint and frame buffer for the 
embryo for the new orientation of the embryo image in the 
present example. The coordination of the distributed 
processing can be performed at client computer 200 by 



19 

application client 210, at server computer 204 by application 
server 220, or by any of the distributed applications 
executing on additional computers, such as 222 and 224. In a 
preferred embodiment, distributed processing is coordinated by 
a program called "VIS" represented by application client 210 
in Fig. 6. 

Other applications of the invention are possible. 
For example, the user can operate a spreadsheet program that 
is being executed by one or more other computer systems 
connected via the network to the user's client computer. 
Once the spreadsheet program has calculated results, those 
results may be sent over the network to the user's client 
computer for display within the hypermedia document on the 
user f s client computer. In this way, computer systems 
located remotely on the network can be used to provide the 
computing power that may be required for certain tasks and to 
reduce the data bandwidth required by only transmitting 
results of the computations. 

Another type of possible application of this 
invention would involve embedding a program which runs only on 
the client machine, but which provides the user with more 
functionality than exists in the hypermedia browser alone. 
An example of this is an embedded client application which is 
capable of viewing and interacting with images which have been 
processed with Dr. Doyle 1 s MetaMAP invention (US Patent 
4,847,604). This MetaMAP process uses object-oriented color"" 
map processing to allow individual color index ranges within 
paletted images to have object identities, and is useful for 
the creation of, for example, interactive picture atlases. It 
is a more efficient means for defining irregular "hotspots" 
on images than the ISMAP function of the World Wide Web, 
which uses polygonal outlines to define objects in images. A 
MetaMAP-capable client-based image browser application can be 
embedded, together with an associated image, within a 
hypermedia document, allowing objects within the 
MetaMAP-processed image to have URL addresses associated with 
them. When a user clicks with a mouse upon an object within 
the MetaMAP-processed image, the MetaMAP client application 



relays the relevant URL back to the hypermedia browser 
application, which then retrieves the HTML file or hypermedia 
object which corresponds to that URL. 

The various processes in the system of the present 
invention communicate through a custom API called 
Mosaic/ External Application Program Interface MEAPI. The 
MEAPI set of predefined messages includes those shown in Table 
I. 

Message Function Message Name 

Messages from server to client: 

1. Server Update Done XtNrefreshNotify 

2 . Server Ready XtNpanelStartNotif y 

3 . Server Exiting XtNpanelExitNotif y 
Messages from client to server: 

4. Area Shown XtNmapNotify 

5. Area Hidden XtNunmapNotify 

6. Area Destroyed XtNexitNotify 

Table I 

The messages in Table I are defined in the file protocol_lib.h 
in Appendix B. The functions of the MEAPI are provided in 
protocol_lib.c of Appendix B. Thus, by using MEAPI a server 
process communicates to a client application program to let 
the client application know when the server has finished 
updating information, such as an image frame buffer, or pixmap 
(Message 1); when the server is ready to start processing 
messages (Message 2) and when the server is exiting or 
stopping computation related to the server application 
program. 

For client to server communications, MEAPI provides 
for the client informing the server when the image display 
window area is visible, when the area is hidden and when the 
area is destroyed. Such information allows the server to 
decide whether to allocate computing resources for, e.g., 
rendering and viewing transformation tasks where the server is 
running an application program to generate new views of a 
multi dimensional object. Source code for MEAPI fundamental 
functions such as handle_client_msg, register_client, 
register_client_msg_callbaek and send_client_msg may be found 
in protocol^ lib.c as part of the source code in Appendix B. 
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Next, a discussion of the software processes that 
perform parsing of a hypermedia document and launching of an 
application program is provided in connection with Table II 
and Figs. 7A, 7B, 8A and 8B. 

Table II, below, shows an example of an HTML tag 
format used by the present invention to embed a link to an 
application program within a hypermedia document. 



< EMBED 

10 TYPE « "type" 

S^-M^SS HREF = "href" 

WIDTH = width 
HEIGHT « height 

> 

TABLE II 
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g As shown in Table II, the EMBED tag includes TYPE, 

aJ HREF, WIDTH and HEIGHT elements. The TYPE element is a 
20 HJ Multipurpose Internet Mail Extensions (MIME) type. Examples 
h of values for the TYPE element are "application/x-vis" or 
£ "video/mpeg" . The type "application /x-vis M indicates that an 
^ application named "x-vis" is to be used to handle the object 
H at the URL specified by the HREF. Other types are possible 
25^ such as "application/x-inventor" , "application/postscript" 

%4 etc. In the case where TYPE is M application/x-vis w this means 
Sj that the object at the URL address is a three dimensional 
^ image object since the program "x-vis" is a data visualization 
tool designed to operate on three dimensional image objects . 
30 However, any manner of application program may be specified by 
the TYPE element so that other types of applications, such as 
a spreadsheet progiram, database program, word processor, etc, 
may be used with the present invention. Accordingly, the 
object reference by the HREF element would be, respectively, a 
35 spreadsheet object, database object, word processor document 
ob j ect , etc . 

On the other hand, TYPE values such as "video/mpeg", 
"image/gif", "video/x-sgi-movie" , etc. describe the type of 
data that HREF specifies. This is useful where an external 
application program, such as a video player, needs to know 
what format the data is in, or where the browser client needs 
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to determine which application to launch based on the data 
format • Thus, the TYPE value can specify either an 
application program or a data type. Other TYPE values are 
possible. HREF specifies a URL address as discussed above for 
5 a data object. Where TYPE is "application/x-vis M the URL 
address specifies a multi-dimensional image object. Where 
TYPE is M video/mpeg M the URL address specifies a video object. 

WIDTH and HEIGHT elements specify the width and 
height dimensions, respectively/ of a Distributed Hypermedia 
10 Object Embedding (DHOE) window to display an external 

application object such as the three dimensional image object 
or video object discussed above. 

Fig. 7 A is a flowchart describing some of the 
^ functionality within the HTMLparse.c file of routines. The 
15 g routines in HTMLparse.c perform the task of parsing a 
hypermedia document and detecting the EMBED tag. In a 
3 J preferred embodiment, the enhancements to include the EMBED 
5 tag are made to an HTML library included in public domain NCSA 
5" Mosaic version 2.4. ^These^f dJ^s^re^^clud^d^as source— code— 
20 7 -"in-i^end-j^^ Note that much 

jf of the source code in Appendix— A - is pre-existing NCSA Mosaic 
~J code. Only those portions of the source code that relate to 

M the new functionality discussed in this specification should 

-0 

be considered as part of the invention. The new functionality 
25 is identifiable as being set off from the main body of source 
code by conditional compilation macros such as M #ifdef . . . 
#endif " as will be readily apparent to one of skill in the 
art. 

In general, the flowcharts in this specification 
30 illustrate one or more software routines executing in a 

computer system such as computer system 1 of Fig. 1. The 
routines may be implemented by any means as is known in the 
art. For example, any number of computer programming 
languages, such as "C M , Pascal, FORTRAN, assembly language, 
35 etc., may be used. Further, various programming approaches 

such as procedural, object oriented or artificial intelligence 
techniques may be employed. 
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The steps of the flowcharts may be implemented by 
one or more software routines, processes, subroutines, 
modules, etc. It will be apparent that each flowchart is 
illustrative of merely the broad logical flow of the method of 
5 the present invention and that steps may be added to, or taken 
away from, the flowcharts without departing from the scope of 
the invention . Further, the order of execution of steps in 
the flowcharts may be changed without departing from the scope 
of the invention. Additional considerations in implementing 
10 the method described by the flowchart in software may dictate 
changes in the selection and order of steps. Some 
considerations are event handling by interrupt driven, polled, 
or other schemes. A multiprocessing or multitasking 
„^ environment could allow steps to be executed "concurrently." 
15 m For ease of discussion the implementation of each flowchart 
Ly may be referred to as if implemented in a single "routine". 
s j* The modifications to NCSA Mosaic version 2.4 

*5 software files HTMLparse.c, HTMLf ormat.c, HTMLwidget.c and 
^ HTML. c will next be discussed, in turn. 
20= Returning to Fig. 7, it is assumed that a hypermedia 

^ document has been obtained at a user's client computer and 
2 that a browser program executing on the client computer 
%J displays the document and calls a first routine in the 
^ HTMLparse.c file called "HTMLparse". This first routine, 
25 HTMLparse, is entered at step 252 where a pointer to the start 
of the document portion is passed. Steps 254, 256 and 258 
represent a loop where the document is parsed or scanned for 
HTML tags or other symbols. While the file HTMLparse.c 
includes routines to handle all possible tags and symbols that 
30 may be encountered, Fig. 7A, for simplicity, only illustrates 
the handling of EMBED tags. 

Assuming there is more text to parse, execution 
proceeds to step 256 where routines in HTMLparse.c obtain the 
next item (e.g., word, tag or symbol) from the document. At 
35 step 258 a check is made as to whether the current tag is the 
EMBED tag. If not, execution returns to step 254 where the 
next tag in the document is obtained. If, at step 258, it is 
determined that the tag is the EMBED tag, execution proceeds 
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to step 260 where an enumerated type is assigned for the tag. 
Each occurrence of a valid EMBED tag specifies an embedded 
object. HTMLParse calls a routine "getjnark" in HTMLparse.c 
to put sections of HTML document text into a "markup 11 text 
5 data structure. Routine get_mark, in turn, calls 

ParseMarkType to assign an enumerated type. The enumerated 
type is an identifier with a unique integer associated with it 
that is used in later processing described below. 

Once all of the hypermedia text in the text portion 
10 to be displayed has been parsed, execution of HTMLparse.c 
routines terminates at step 262. 

Fig. 7B is a flowchart of routines in file 
HTMLformat.c to process the enumerated type created for the 
EMBED tag by routines in HTMLparse.c. In the X-Window 
15 rg implementation of a preferred embodiment, the enumerated type 
UJ is processed as if it is a regular Motif /XT widget. For 

details on X-Window development see, e.g., "Xlib Programming 
5 Manual," "X Toolkit Intrinsics Programming Manual" and "Motif 
Programming Manual" published by O'Reilly & Associates, Inc. 
20 HTMLformat is entered at step 270 where a pointer to the 
enumerated type to process is passed. 
(I At step 272 the parameters of the structure are 

4 initialized in preparation for inserting a DrawingArea widget 
% on an HTML page. This includes providing values for the width 
25 and height of a window on the display to contain an image, 

position of the window, style, URL of the image object, etc. 
Various codes are also added by routines in HTMLformat.c (such 
as TriggerMarkChanges) to insert an internal representation of 
the HTML statement into an object list maintained internally 
30 by the browser. In the X-Window application corresponding to 
the source code of Appendix A, the browser is NCSA Mosaic 
version 2.4. 

Fig. 8A is a flowchart for routine HTMLwidget. 
HTMLwidget creates display data structures and launches an 

35 external application program to handle the data object 

i 

specified by the URL in the EMBED tag. 

HTMLwidget is entered at step 280 after HTMLformat 
has created the internal object representation of the EMBED 




tag. HTMLwidget is passed the internal object and performs 
its processing on the object. At step 282 the DrawingArea 
widget is created according to the type of the internal 
representation, from HTMLf ormat, specified in the internal 
object. Similarly, at step 284 a pixmap area for backing 
storage is defined. 

At step 286 a check is made as to whether the type 
attribute of the object, i.e., the value for the TYPE element 
of the EMBED tag, is an application. If so, step 290 is 
executed to launch a predetermined application. In a 
preferred embodiment an application is launched according to a 
user-defined list of application type/application pairs. The 
list is defined as a user-configurable XResource as described 
in M Xlib Programming Manual." An alternative embodiment could 
use the MIME database as the source of the list of application 
type/ application pairs. The routine 

"vis_start_ exter Misapplication" in file HTMLf ormat . c is 
invoked to match the application type and to identify the 
application to launch. 

The external application is started as a child 
process of the current running process (Mosaic) , and informed 
about the window ID of the DrawingArea created in HTMLf ormat. 
The external application is also passed information about the 
ID of the pixmap, the data URL and dimensions. Codes for 
communication such as popping-up/iconifying, start 
notification, quit notification and refresh notification with 
external applications and DrawingArea refreshing are also 
added. Examples of such codes are (1) "setup/ start" in 
vis_register_client and vis_get_j>anel_window in HTMLwidgets . c ; 
(2) "handle messages from external applications" in 
vis_handle_j?anel_msg in HTMLwidgets . c ; (3) "send messages to 
external applications" in vis_send_msg in HTMLwidgets . c ; (4) 
"terminate external applications" in vis_exit in HTMLwidgets . c 
which calls vis_send_msg to send a quit message; and (5) 
"respond to refresh msgs" in vis_redraw in HTMLwidgets . c . 

If, at step 286, the type is determined not to be an 
application object (e.g., a three dimensional image object in 
the case of application "x-vis") a check is made at step 288 



to determine if the type is a video object. If so, step 292 
is executed to launch a video player application. Parameters 
are passed to the video player application to allow the player 
to display the video object within the DrawingArea within the 
display of the portion of hypermedia document on the clients 
computer. Note that many other application objects types are 
possible as described above. 

Fig. 8B is a flowchart for routine HTML. Routine 
HTML takes care of "shutting down" the objects, data areas, 
etc. that were set up to launch the external application and 
display the data object. HTML is entered at step 300 and is 
called when the display or other processing of the EMBED tag 
has been completed. At step 302 the display window is removed 
and the memory areas for the pixmap and internal object 
structure is made free for other uses. Completion of 
processing can be by user command or by computer control. 

The present invention allows a user to have 
interactive control over application objects such as three 
dimensional image objects and video objects. In a preferred 
embodiment, controls are provided on the external 
applications 1 user interface. In the case of a VIS/panel 
application, a process, "panel" creates a graphical user 
interface (GUI) thru which the user interacts with the data. 
The application program, VIS, can be executing locally with 
the user's computer or remotely on a server, or on one or more 
different computers, on the network. The application program 
updates pixmap data and transfers the pixmap data (frame image 
data) to a buffer to which the browser has access. The 
browser only needs to respond to the refresh request to copy 
the contents from the updated pixmap to the 
DrawingArea. The Panel process sends messages as "Msg" 
sending performed by routines such as vis_send_msg and 
vis_handle_panel_msg to send events (mousemove, keypress, 
etc.) to the external application. 

Fig. 9 is a screen display of the invention showing 
an interactive application object (in this case a three 
dimensional image object) in a window within a browser window. 
In Fig. 9, the browser is NCSA Mosaic version 2.4. The 



processes VIS, Panel and VRServer work as discussed above* 
Fig. 9 shows screen display 356 Mosaic window 350 containing 
image window 352 and a portion of a panel window 354. Note 
that image window ,352 is within Mosaic window 350 while panel 
window 354 is external to Mosaic window 350. Another 
possibility is to have panel window 354 within Mosaic window 
350. By using the controls in panel window 354 the user is 
able to manipulate the image within image window 352 in real 
time do perform such operations as scaling, rotation, 
translation, color map selection, etc. In Fig. 9, two Mosaic 
windows are being used to show two different views of an 
embryo image. One of the views is rotated by six degrees from 
the other view so that a stereoscopic effect can be achieved 
when viewing the images. Communication between Panel and VIS 
is via "Tooltalk" described in, e.g., "Tooltalk 1.1.1 
Reference Manual, n from SunSoft. 

Fig. 10 is an illustration of the processes VIS, 
Panel and VRServer discussed above. As shown in Fig. 10, the 
browser process, Mosaic, communicates with the Panel process 
via inter-client communication mechanisms such as provided in 
the X-Window environment. The Panel process communicates with 
the VIS process through a communications protocol (ToolTalk, 
in the preferred embodiment) to exchange visualization command 
messages and image data. The image data is computed by one or 
more copies of a process called VRServer that may be executing 
on remote computers on the network. VRServer processes 
respond to requests such as rendering requests to generate 
image segments. The image segments are sent to VIS and 
combined into a pixmap, or frame image, by VIS. The frame 
image is then transferred to the Mosaic screen via 
communications between VIS, Panel and Mosaic* A further 
description of the data transfer may be found in the paper 
"Integrated Control of Distributed Volume Visualization 
Through the World-Wide -Web, " referenced above. 

In the foregoing specification, the invention has 
been described with reference to a specific exemplary 
embodiment thereof. It will, however, be evident that various 
modifications and changes may be made thereunto without 
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departing from the broader spirit and scope of the invention 
as set forth in the appended claims. For example, various 
programming languages and techniques can be used to implement 
the disclosed invention. Also, the specific logic presented 
5 to accomplish tasks within the present invention may be 

modified without departing from the scope of the invention. 
Many such changes or modifications will be readily apparent to 
one of ordinary skill in the art. The specification and 
drawings are, accordingly, to be regarded in an illustrative 
10 rather than a restrictive sense, the invention being limited 
only by the provided claims. 
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WHAT IS C LAIMED IS : 



IX A method for running an application program in 
a computer network environment, comprising: 

providing at least one client workstation and one 
network serves coupled to said network environment, wherein 
said network environment is a distributed hypermedia 
environment ; 

displaying, on said client workstation, at least a 
portion of a hypermedia document received over said network 
from said server A wherein said hypermedia document includes an 
embedded controllable application; and 

interactively controlling said embedded controllable 
application from said client workstation via communications 
sent over said distributed hypermedia environment. 
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2. The method of claim 1, wherein the step of 
displaying is performed by using a hypermedia browser 
application. 



17 3 . The methbd of claim 2 , wherein instructions for 

18 ~: controlling said embedded controllable application reside on 

19 \j said network server, wherein said step of interactively 

20 ^ controlling said embedded controllable application includes 

21 the following substeps: 

22 issuing, from tfoe client workstation, one or more 

23 commands to the network seVver; 

24 executing, on the network server, one or more 

25 instructions in response ta said commands; 

26 sending information from said network server to said 

27 client workstation in response to said executed instructions; 

28 and 

29 processing said information at the client 

30 workstation to interactively <\ontrol said embedded 

31 controllable application. 
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4 . T|he method of claim 2 , wherein instructions for 
controlling saidi embedded controllable application reside on 
said client workstation. 

5* Thte method of claim 2, wherein the 
communications to \ interactively control said embedded 
controllable application from said client workstation continue 
to be exchanged between the controllable application and the 
hypermedia browser even after the controllable application 
program has been launched. 

6. The method of claim 3, wherein said embedded 
controllable application is a multi-dimensional viewer, 

7. The method of claim 3, wherein said embedded 
controllable application is a spreadsheet program. 

8. The method of claim 3, wherein said embedded 
controllable application is a database program. 

9. The methoa of claim 3, wherein said embedded 
controllable application \s a word processor. 

10. lftie method of claim 3, wherein said substeps of 
issuing and sending are via an open protocol. 

11. The\ method of claim 10 , wherein said open 
protocol is an International Standards Organization (ISO) 
protocol . 



12. The method of claim 11, wherein said ISO 
protocol is Transfer Cdntrol Protocol/ Internet Protocol 
(TCP/IP) and said network is the Internet. 



13. The methodXof claim 12, wherein HyperText 
Transfer Protocol is used Vo transfer said hypermedia document 
between said client workstation and said server. 
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14. \The method of claim 13, wherein Hyper Text 
Markup Language\is used to specify said embedded controllable 
application within said hypermedia document. 

15. A method for running an application program in 
a computer network \environment , comprising: 

providingXat least one client workstation and one 
network server coupled to said network environment, said 
network including a plurality of general purpose workstations, 
wherein said network environment is a distributed hypermedia 
environment ; 

displaying, on said client workstation, at least a 
portion of a hypermedia Wocument received over said network 
from said server, wherein said hypermedia document includes at 
least a first embedded multi-dimensional data visualization 
application; and 

interactively cdntrolling said embedded multi- 
dimensional data visualization application from said client 
workstation via communications sent over said distributed 
hypermedia environment wherlein data image rendering is 
performed by said plurality! of general purpose workstations 
using distributed processing. 
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16. Th£ methodVof claim 15, wherein the step of 
displaying is performed by\using a hypermedia browser 
application. 



84 17. Thi 

85 dimensional data 



method of claim 15, wherein the multi- 
Lsualization includes volume visualization. 



86 18. The mqthod of claim 15, wherein the multi- 

87 dimensional data visualization includes two dimensional image 

88 processing. 



89 19. The method\of claim 15, wherein the multi- 

90 dimensional data visualization includes image analysis. 
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20 A The method of claim 15, wherein the multi- 
dimensional data visualization includes the display of 
animated sequences. 

21. The method of claim 15, wherein the multi- 
dimensional data visualization includes a geometric data 
viewer to display computer aided design files. 

22. The method of claim 15, wherein the multi- 
dimensional data visualization includes displaying molecular 
modeling data. \ 

23. Y he **etRx>d of claim 15, wherein a hypermedia 
browser is executing on\the client workstation, wherein 
communications to interactively control said embedded 
controllable application from said client workstation continue 
to be exchanged between the controllable application and the 
hypermedia browser Wen after the controllable application 
program has been launched. 

24. A method for interactively controlling an 
embedded object in a document displayed on a client computer, 
wherein the client computer includes a processor coupled to a 
display device and to avuser input device, wherein the 
processor is further coupled to a computer network, wherein 
the computer network is coupled to a server computer and one 
or more additional computers, wherein the server computer 
includes a local storage device containing a document, wherein 
the document includes an embedded object, wherein an 
application program for manipulating the embedded object 
resides on a first additional^ computer, the method comprising 
the following steps: \ 

transferring, over the network, at least a portion 
of the document from the serven computer to the client 
computer ; \ 

accepting first signals from the user input device 
that indicate that the embedded object is to be manipulated; 



J 





33 

124 issuing bommands from the client computer to the 

125 first additional computer in response to the first signals; 

126 executing \ by using the first additional computer, 

127 instructions in the Application program in response to the 

128 issued commands, wherein the executed instructions generate 

129 information about manipulating the embedded object; 

130 communicating \ via the network, the information 

131 about manipulating the embedded object from the first 

132 additional computer to the\ client computer; and 

133 using the client computer to manipulate the embedded 

134 object according to the communicated information. 




143 

144= 

145 
146 
147 

148 
149 



25. Thevmethod of claim 24, wherein said document 
a hypermedia document. 

26. \he method of claim 24, further comprising the 
^teps of executing instructions in a second application 
program on a second additional computer in response to the 
issued commands, wherein the instructions executed by the 
second additional computer result in information about 
manipulating the embedded object being generated more quickly. 

v \ 

27. The method of claim 26, wherein said document 
i$/-a hypermedia document. 




28. TheVmethod of claim 26, wherein the embedded 
/ object is a multi-dimensional image displayable in any of a 

plurality of orientations. 

\ \ 

29. The method of claim 28, wherein said document 
is a hypermedia document. 



150 30. The method of claim 28, wherein the executed 

151 instructions perform three dimensional display transformations 

152 to determine the seconcK orientation of the multi-dimensional 

153 image object. 



31. \ The method of claim 30, wherein said document 
is a hypermedia document. 



32. Ittie method of claim 28, wherein the executed 
instructions perform image rendering to determine an 
orientation of the multi-dimensional image. 

33. The taethod of claim 32, wherein said document 
is a hypermedia document. 



14. A method for displaying a three dimensional 
image object on a client computer, wherein the client computer 
includes a processor coupled to a display device, wherein the 
processor is\further coupled to a computer network, wherein 
the computer network is coupled to a server computer and one 
or more additional computers, wherein the server computer 
includes a loc&L storage device containing a hypermedia 
document, wherein the hypermedia document includes a three 
dimensional imagte object embedded within the hypermedia 
document, wherein the three dimensional image object is 
displayable in a plurality of orientations, the method 
comprising the following steps: 

transferring, over the network, at least a portion 
of the hypermedia document from the server computer to the 
client computer; 

displaying \ on the display device, by using the 
processor, at least apportion of the hypermedia document, 
wherein the displayed mortion of the hypermedia document 
includes the three dimensional image object displayed in a 
first orientation; 

using the client computer to issue commands over the 

network ; 

executing instruction on a first additional computer 
in response to the issued\ commands, wherein the executed 
instructions determine a second orientation for display of the 
three dimensional image obiect; 
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187 communicating, via the network, information about 

188 the second orientation from the first additional computer to 

189 the client computer; Vnd 

190 using the cMent computer to redisplay the three 

191 dimensional image obj eat in the second orientation. 

192 35. \ The method of claim 34, wherein said network is 

193 a distributed hypermedia environment. 

194 36. The method of claim 34, further comprising the 

195 steps of executing instructions on a second additional 

196 computer in response to the issued commands, wherein the 

197 instructions executed by the second computer enable the second 

198 orientation to be determined more quickly than when only the 

1993 first additional computer executes instructions. 

vy \ 

20Q\j 37. The method of claim 36, wherein said network is 

201 ^ a distributed hypermedia environment. 

202 j 38. The methbd of claim 36, wherein the executed 

2037^, instructions perform volume rendering to determine the second 

204 Q orientation of the three\ dimensional image object. 

205 A 39. The method \of claim 38, wherein said network is 

206 C a distributed hypermedia environment. 

207 40. The method off claim 36, wherein the executed 

208 instructions perform three dimensional display transformations 

209 to determine the second orientation of the three dimensional 

210 image object. \ 

211 41. The method of claim 40, wherein said network is 

212 a distributed hypermedia environment. 

213 42. The method of claim 34, wherein the client 

214 computer includes a user input device coupled to the 

215 processor, the method further comprising the following steps: 
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216 accept ing\ signals from the user input device, 

217 wherein the accepted signals indicate that the second 

218 orientation is to be idetermined . 

219 43 • The methbd of claim 42, wherein said 

220 network is a distributed hypermedia environment. 
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EMBEDDED PROGRAM OBJECTS IN 
DISTRIBUTED HYPERMEDIA SYSTEMS 

ABSTRACT OF THE DISCLOSURE 
5 A system allowing a user of a browser program on a 

computer connected to an open distributed hypermedia system to 
access and execute an embedded program object. The program 
object is embedded into a hypermedia document much like data 
objects. The user may select the program object from the 
10 screen. Once selected the program object executes on the 

user's (client) computer or may execute on a remote server or 
additional remote computers in a distributed processing 
arrangement. After launching the program object, the user is 
^ able to interact with the object as the invention provides for 
15 y ongoing interprocess communication between the application 
^ object (program) and the browser program. One application of 
the embedded program object allows a user to view large and 
complex multi-dimensional objects from within the browser's 
7" window. The user can manipulate a control panel to change the 
20 H viewpoint used to view the image. The invention allows a 
rf program to execute on a remote server or other computers to 
j calculate the viewing transformations and send frame data to 
-43 the client computer thus providing the user of the client 
computer with interactive features and allowing the user to 
25 have access to greater computing power than may be available 
at the user's client computer. 
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